ci: optimize sparse-checkout and cache to reduce cache size#26692
ci: optimize sparse-checkout and cache to reduce cache size#26692
Conversation
Exclude additional large files from CI checkout that are not needed for builds/tests: - docs/images/ (~90MB) - Documentation images - packages/app-store/*/static/*.png,jpg,jpeg,gif (~140MB) - App store screenshots These files are only needed for documentation rendering and app store UI display, not for CI builds, linting, type checking, or tests. SVG icons in app-store are preserved as they may be needed for the build. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
The apps/web/public/app-store directory contains ~151MB of static files copied from packages/app-store/*/static/ during build. These files are regenerated during yarn install and don't need to be cached. Combined with the sparse-checkout exclusions, this should significantly reduce the git checkout cache size. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The first checkout in the prepare job was doing a full checkout (~535 MB), and then dangerous-git-checkout applied sparse-checkout. But the files from the first checkout remained on disk because sparse-checkout doesn't remove files that were already checked out. By limiting the first checkout to only .github (which is needed to access the local actions), we avoid downloading the full repo twice. The actual sparse-checkout with exclusions is then applied by dangerous-git-checkout. This should reduce the cache from ~504 MB to ~148 MB. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
E2E results are ready! |
Devin AI is resolving merge conflictsThis PR has merge conflicts with the Devin will:
If you prefer to resolve conflicts manually, you can close the Devin session and handle it yourself. |
Resolved merge conflicts in: - .github/actions/cache-checkout/action.yml: Kept PR's save/restore mode design - .github/workflows/pr.yml: Kept PR's sparse-checkout + dangerous-git-checkout approach Co-Authored-By: unknown <>
Devin AI is resolving merge conflictsThis PR has merge conflicts with the Devin will:
If you prefer to resolve conflicts manually, you can close the Devin session and handle it yourself. |
|
Merge conflicts have been resolved. The conflict in The |
Co-Authored-By: unknown <>
…fix pr.yml compatibility Co-Authored-By: unknown <>
305c86d to
cbfb3c0
Compare
|
I've corrected the merge conflict resolution. The previous resolution incorrectly kept the separate save/restore approach, but I've now properly resolved it to:
The |
…6692) * ci: optimize sparse-checkout to reduce cache size by ~230MB Exclude additional large files from CI checkout that are not needed for builds/tests: - docs/images/ (~90MB) - Documentation images - packages/app-store/*/static/*.png,jpg,jpeg,gif (~140MB) - App store screenshots These files are only needed for documentation rendering and app store UI display, not for CI builds, linting, type checking, or tests. SVG icons in app-store are preserved as they may be needed for the build. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: exclude generated app-store static files from cache The apps/web/public/app-store directory contains ~151MB of static files copied from packages/app-store/*/static/ during build. These files are regenerated during yarn install and don't need to be cached. Combined with the sparse-checkout exclusions, this should significantly reduce the git checkout cache size. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: fix sparse-checkout by limiting first checkout to .github only The first checkout in the prepare job was doing a full checkout (~535 MB), and then dangerous-git-checkout applied sparse-checkout. But the files from the first checkout remained on disk because sparse-checkout doesn't remove files that were already checked out. By limiting the first checkout to only .github (which is needed to access the local actions), we avoid downloading the full repo twice. The actual sparse-checkout with exclusions is then applied by dangerous-git-checkout. This should reduce the cache from ~504 MB to ~148 MB. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: update cache-checkout to include sparse-checkout exclusions and fix pr.yml compatibility Co-Authored-By: unknown <> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
What does this PR do?
Reduces the CI git checkout cache size from ~637 MB to ~86 MB - an 87% reduction.
Root Cause
The first
actions/checkout@v4in the prepare job was doing a full checkout (~535 MB), thendangerous-git-checkoutapplied sparse-checkout. But files from the first checkout remained on disk because sparse-checkout doesn't remove files that were already checked out.Fix
.githubonly - The first checkout only needs access to local actions, so we usesparse-checkout: .githubto avoid downloading the full repodangerous-git-checkoutexcludes large files not needed for CI:docs/images/(~90 MB)packages/app-store/*/static/*.{png,jpg,jpeg,gif}(~101 MB)apps/web/public/app-storefrom cache (generated files)Results
Mandatory Tasks (DO NOT REMOVE)
How should this be tested?
Human Review Checklist
.githubdirectory to access local actions.svgextension, not excluded)Requested by: @keithwillcode
Link to Devin run: https://app.devin.ai/sessions/762cef383ff64e60a5dbd356d6d5759d